Conventional cameras provide photographs of real world scenes with a limited field of view of the scene being photographed. In many scenarios, the photographer desires an image corresponding to a wider field of view. Typically, the photographer can resort to two methods of generating a wide field of view image. The first method is to capture the wide field of view image directly; e.g., with a wide-angle lens, or with a specialized system of mirrors to reflect the wide field of view onto the sensor. The second method is to capture a collection of images, each image having a narrower field of view, and then use one of a variety of digital image stitching techniques to combine the narrow field of view images into a composite digital image. The composite digital image will appear to be a single wide field of view image.
When a camera captures an image of a scene, the image represents a perspective projection of the scene onto the planar sensor. Inherent to perspective projection is a natural distortion, namely, objects closer to the center of the image appear smaller than similar objects near the edges of the image. This distortion becomes immediately apparent when attempting to stitch subsequent images together. Therefore, typical image stitching systems include a step of warping the images to compensate for this perspective distortion. In a physical sense, the perspective distortion would not exist if the sensor were not planar, but rather spherical (with the radius of the sphere depending on the focal length of the lens). In scenarios where the sequence of images to be stitched is captured by rotating a camera on a tripod (or rotating a camera about a vertical axis), the perspective distortion would not exist in the horizontal direction if the sensor were cylindrical (with the radius of the cylinder depending on the focal length of the lens, and the axis of the cylinder lying on the axis of rotation of the camera). Even though there would still be distortion in the vertical direction of the images, this distortion would not hamper the photographer's ability to seamlessly stitch together such a sequence of images.
Since it is extremely difficult and expensive to manufacture sensors that are spherical or cylindrical in shape, compensation for the perspective distortion is generally performed after the image has been captured. The compensation is performed by geometrically warping the image so that it appears to have been captured on the spherical or cylindrical sensor. In the article “Panoramic Stereo Imaging System with Automatic Disparity Warping and Seaming” by H.-C. Huang and Y.-P. Hung (Graphical Models and Image Processing, Vol. 60, No. 3, May, 1998, pp. 196–208), the authors derive the equations relating pixels of a cylindrical sensor to that of a planar sensor. The derivation of the equations relating pixels of a spherical sensor to that of a planar sensor is similar. For the spherical sensor, the pixel (x,y) of the compensated image Ĩ is related to the captured image I by the relationship:Ĩ(x,y)=I(f tan(xpxf−1)/px,f tan(ypyf−1)/py),where px and py are the horizontal and vertical pixel sizes, respectively, f is the focal length, and (x,y)=(0,0) corresponds to the center of the image. For the cylindrical sensor, the pixel (x,y) of the compensated image Ĩ is related to the captured image I by the relationship:Ĩ(x,y)=I(f tan(xpxf−1)/px,yf tan(xpxf−1)/xpx), for x≠0, andĨ(0,y)=I(0,y).
After each image in the sequence has been geometrically warped, typical image stitching systems then determine the parameters that optimally align the set of images (for example, by cross correlation or phase correlation, or by knowledge of the geometry of the camera at each capture position). Once the images are aligned, they are blended together (by taking weighted averages of overlapping pixels, for example) to form a composite digital image. Finally, depending on the choice of output, the composite digital image can be again geometrically warped, this time to simulate a perspective projection of the wide field of view scene onto a chosen reference planar sensor.
In some image stitching systems, specifically systems that construct composite digital images in real time, or systems that construct a large sequence of composite digital images (e.g., a system that stitches together images from video sequences to form a composite video sequence), the step of geometrically warping the images to compensate for the perspective distortion requires a significantly large portion of the total computational time of the system. Therefore, any mechanism that would alleviate the need to perform geometric warping of the images would remove this bottleneck in real-time or video image stitching systems.
Another type of distortion that occurs in most camera systems (especially those with wide-angle lenses) is lens distortion. Lens distortion frequently manifests itself as a radial distortion, where objects further from the center of the image appear smaller than those near the center of the image. In addition, lens irregularities and aberrations can induce local distortions in different areas of the image plane.
A method exists in the art to compensate for lens distortion without geometrically warping the images after they have been captured. U.S. Pat. No. 5,489,940, “Electronic Imaging System and Sensor for Correcting the Distortion in a Wide-Angle Lens”, and U.S. Pat. No. 5,739,852, “Electronic Imaging System and Sensor for Use Therefor with a Nonlinear Distribution of Imaging Elements”, both by C. Richardson and B. Stuckman, describe an imaging system comprising a sensor with a nonlinear distribution of sensor elements, wherein the distribution of the imaging elements corrects for the distortion in a wide angle lens. More specifically, the distribution of sensor elements has a relatively low density at a center point of the sensor surface and a relatively high density along the periphery of the sensor surface. However, neither of these patents are directly applicable to systems compensating for perspective distortion. Perspective distortion, as discussed previously, can be compensated for by projecting the image onto a nonplanar surface. Lens distortion, in the method of the two aforementioned patents, is compensated by projecting the image through a nonlinear function. This nonlinear function is selected such that the scene appears to be projected onto a planar surface, as expected by perspective projection. However, the relative densities of the distribution of sensor elements near the center and periphery of the image are inversely related to what the relative densities should be to compensate for perspective distortion. Consequently, when using digital stitching techniques to combine multiple images captured from the type of sensor disclosed in these patents, a geometric warping must still be applied to overcome the perspective projection.
Therefore, there exists a need in the art for an imaging system that would alleviate the need to perform geometric warping of images to compensate for perspective distortion after the images have been captured.